Skip to content

HADOOP-18782 upgrade to snappy-java 1.1.10.1 due to CVEs#5773

Merged
steveloughran merged 1 commit intoapache:trunkfrom
pjfanning:HADOOP-18782-snappy
Jun 27, 2023
Merged

HADOOP-18782 upgrade to snappy-java 1.1.10.1 due to CVEs#5773
steveloughran merged 1 commit intoapache:trunkfrom
pjfanning:HADOOP-18782-snappy

Conversation

@pjfanning
Copy link
Member

@pjfanning pjfanning commented Jun 24, 2023

Description of PR

upgrade to snappy-java 1.1.10.1 due to CVEs

How was this patch tested?

CI build

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@slfan1989
Copy link
Contributor

slfan1989 commented Jun 24, 2023

+1（binding) wait jenkins compilation result.

@hadoop-yetus
Copy link

hadoop-yetus commented Jun 25, 2023

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 58s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 2s codespell was not available.
+0 🆗 detsecrets 0m 2s detect-secrets was not available.
+0 🆗 xmllint 0m 2s xmllint was not available.
+0 🆗 shelldocs 0m 2s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 18m 1s Maven dependency ordering for branch
+1 💚 mvninstall 24m 33s trunk passed
+1 💚 compile 19m 6s trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 compile 16m 51s trunk passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 mvnsite 21m 37s trunk passed
+1 💚 javadoc 9m 2s trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 7m 33s trunk passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 shadedclient 40m 35s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 33s Maven dependency ordering for patch
+1 💚 mvninstall 20m 11s the patch passed
+1 💚 compile 18m 6s the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 javac 18m 6s the patch passed
+1 💚 compile 16m 56s the patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 javac 16m 56s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 15m 9s the patch passed
+1 💚 shellcheck 0m 1s No new issues.
+1 💚 javadoc 8m 48s the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 7m 30s the patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 shadedclient 40m 50s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 775m 30s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 40s The patch does not generate ASF License warnings.
1036m 15s
Reason Tests
Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler
hadoop.mapreduce.v2.TestMRJobs
hadoop.mapreduce.v2.TestUberAM
hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2
hadoop.hdfs.server.namenode.ha.TestObserverNode
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5773/1/artifact/out/Dockerfile
GITHUB PR #5773
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname Linux 33a7ae8dcf6a 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0f8ed61
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5773/1/testReport/
Max. process+thread count 3094 (vs. ulimit of 5500)
modules C: hadoop-project . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5773/1/console
versions git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn approved these changes Jun 26, 2023
View reviewed changes
Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM,
failing tests are failing continuously, can ignore for now, need to fix them soon....

@steveloughran
Copy link
Contributor

steveloughran commented Jun 27, 2023

failing tests are failing continuously, can ignore for now, need to fix them soon....

any underlying cause?

steveloughran
steveloughran reviewed Jun 27, 2023
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
+1

@steveloughran steveloughran merged commit 56ef05a into apache:trunk Jun 27, 2023
@ayushtkn
Copy link
Member

ayushtkn commented Jun 27, 2023

broken in the daily build, since months
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1268/#showFailuresLink
they are broken since 3-4 months, I chased some earlier MR related they were Junit upgrade related, but these are different.

it is expecting to log the thread dump in the file, but that isn't getting dumped now

java.lang.AssertionError: No thread dump
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1273)

someone removed it or something broke in that logic, haven't got chance to debug yet....

@pjfanning pjfanning deleted the HADOOP-18782-snappy branch June 27, 2023 11:11
@steveloughran
Copy link
Contributor

steveloughran commented Jun 27, 2023

looks like a really brittle test, including comments that don't match the code, and a text file scan which (a) doesn't log the output and (b) could well be brittle against Java versions changing the format of the output.

asfgit pushed a commit that referenced this pull request Jun 27, 2023
@pjfanning @steveloughran
HADOOP-18782. Upgrade to snappy-java 1.1.10.1 due to CVEs (#5773)
384891c 
Addresses CVE-2023-34454

Contributed by PJ Fanning
@ayushtkn
Copy link
Member

ayushtkn commented Oct 2, 2023

These tests are fixed now: https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1367/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/

The culprit was:
HADOOP-18649

And this was committed in march & these tests were also failing since 4-5 months, now post revert they are passing, so hopefully we are sorted :-)

jiajunmao pushed a commit to jiajunmao/hadoop-MLEC that referenced this pull request Feb 6, 2024
@pjfanning @jiajunmao
HADOOP-18782. Upgrade to snappy-java 1.1.10.1 due to CVEs (apache#5773)
8beab83 
Addresses CVE-2023-34454

Contributed by PJ Fanning
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steveloughran steveloughran steveloughran left review comments

@ayushtkn ayushtkn ayushtkn approved these changes

@slfan1989 slfan1989 slfan1989 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@pjfanning @slfan1989 @hadoop-yetus @steveloughran @ayushtkn